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Executive Summary 



The use of student data systems to improve education and help students succeed is a national 
priority. The Elementary and Secondary Education Act , as reauthorized in 2002, calls for the 
collection, analysis, and use of student achievement data to improve school outcomes. Data 
systems are expected to play an integral role in improving educational decision making at all 
levels — including that of the classroom teacher. The U.S. Department of Education has provided 
support for major improvements in the quality of state data systems to enable longitudinal 
analysis of student data and linkage between student outcomes and other education system 
variables. These improved systems are supporting educational research and decision making at 
the state level, but at the local level, district and school staff work with district rather than state 
data systems. If data-driven decision making is to become an effective tool for improving the 
instruction provided to students, policymakers need a clear understanding of these data systems 
used at the local level and of the decision-making processes in schools and districts. 

Since 2006 the national Study of Education Data Systems and Decision Making, sponsored 
by the U.S. Department of Education’s Policy and Program Studies Service, has been examining 
both the implementation of student data systems per se and the broader set of practices involving 
the use of data to improve instruction, regardless of whether or not the data are stored in and 
accessed through an electronic system. The study’s data collections included a national survey of 
districts in spring 2007 and site visits during school years 2006-07 and 2007-08 to a purposive 
sample of districts and schools selected on the basis of their active involvement in the use of data 
for instructional improvement. The study team also conducted secondary analyses of national 
teacher survey responses to questions concerning data system access and use. 

Earlier study reports have documented a dramatic increase in the proportion of teachers with 
access to a student data system between 2005 and 2007 and described school practices with 
respect to data use and the challenges that are part of student data system implementation. This 
final report builds on the picture of local practices in implementing data-driven decision making 
provided in the earlier reports by presenting data from the national district survey as well as from 
site visits conducted during 2007-08 at 36 schools in 12 districts. 

Key findings from surveys and site visits with respect to district data systems and strategies 
for supporting the use of data in instructional decision making are described below. 

District Data Systems and Use of Data for Decision Making 

Districts are still in the process of building their data system technology capacity. An 

examination of district capacity with respect to data systems needs to take into account the 
multiple types of systems containing data concerning students and other aspects of the education 
system (see Exhibit ES-1). Nearly all school districts have an electronic student information 
system providing real-time access to information such as enrollment and attendance. According 
to district survey respondents, the majority of districts (70 percent) have had this type of system 
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for six or more years (see Exhibit ES-2). More recently, districts are acquiring other types of 
electronic data systems: 79 percent report having an assessment system that organizes and 
analyzes benchmark assessment data, 77 percent report having a data warehouse that provides 
access to current and historical data on students as well as data on other aspects of district 
functioning, and 64 percent report having an instructional or curriculum management system to 
support access to curriculum and instructional resources. 



Exhibit ES-1. Types of Electronic Student Data Systems 

1. Student information systems provide real-time access to student data such as attendance, 
demographics, test scores, grades and schedules. 

2. Data warehouses are electronic data collection and storage systems that provide access to 
current and historical data on students, personnel, finances and so on. 

3. Instructional or curriculum management systems provide a unifying framework to support 
access to curriculum and instructional resources such as planning tools, model lesson 
plans, creation of benchmark assessments, linkage to state content or performance 
standards, communication and collaboration tools (e.g., threaded discussion forums). 

4. Assessment systems support rapid organization and analysis of benchmark assessment 
data. 

Source: Wayman (2005). 



Exhibit ES-2. Districts With Electronic Student Data Systems, by Type 
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Exhibit reads: In 2007-08, more than 99 percent of districts reported 
having a student information system. 

Source: 2007-district survey question 4. 
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The district survey found no significant relationships between district size or the proportion 
of a district’s students living in poverty and the likelihood of having a particular type of student 
data system. 1 The few districts without any electronic data system (10 out of 427 districts in the 
survey sample) tended to be smaller in size than other districts in the survey sample but similar 
in terms of the percentage of students in poverty and the percentage of their schools that are in 
Title I or not making AYP. 

At this point in time, districts are looking for a way to effectively link their multiple 
data systems since there is no “single solution” data system and there is no simple recipe 
for effective system implementation. Most districts have multiple, distinct data systems. The 
number of electronic data systems being used to support decisions about instruction in the case 
study districts ranged from three to seven. Although not a problem in principle, the use of 
multiple systems can be a problem in practice. On the district survey, over 60 percent of districts 
reported that lack of interoperability across data systems was a current barrier to expanded use of 
data-driven decision making. 

Districts’ initial acquisition of data systems and use of data has been driven by 
accountability requirements. The Elementary and Secondary Education Act requirements that 
states and districts report on the progress of every student subgroup toward academic proficiency 
and close the proficiency gap among student subgroups have both motivated districts to acquire 
data systems and shaped the nature of those systems. Districts are much more likely to have 
electronic systems with data such as student demographics and test scores than to have the ability 
to combine data from different types of systems or to link instructional resources to achievement 
data. Over 90 percent of the districts surveyed reported having electronically stored data on 
student demographics and attendance, student grades, student test scores on statewide 
assessments, and student course enrollment histories. In contrast, less than half of districts have 
electronic data systems that allow them to link outcomes to processes as required for 
continuous improvement. For example, only 42 percent of districts can generate data reports 
showing student performance linked to participation in specific instructional programs, and just 
over a third (38 percent) can execute queries concerning student performance linked to teacher 
characteristics. 

To support better decisions about instruction, data systems should make available data on the 
same student or group of students over time and support looking at the performance of students 
with different educational experiences (i.e., different teachers or instructional programs). Exhibit 
ES-3 shows the percentage of districts reporting that their systems include such tools for data 
use. 



For the remainder of this report, the percentages reported will be for districts indicating that they have an 
electronic student information data system (which is greater than 99 percent of all surveyed districts) unless 
otherwise specified. 



XI 




Exhibit ES-3. District Data System Query Capabilities in 2007-08 



Type of Query 


Percent of 
Districts 
With This 
System 
Capability 


Percent of 
Students 
Represented 
by These 
Districts 


Individual student history overtime (e.g., cumulative grades)* 


83 


88 


Drill-down capability (ability to query a school-level finding to efficiently 
examine a subset of data at the grade, classroom or student level) 


76 


85 


Individual student assessment performance over time** 


72 


85 


Student performance linked to specific teachers** 


67 


78 


Student performance linked to specific instructional programs 


42 


50 


Student performance linked to teacher information or characteristics 


38 


45 



Exhibit reads: In 2007-08, 83 percent of districts had electronic data systems that had the capacity to 
support queries about individual student histories overtime. These districts serve approximately 
88 percent of the nations’ public school students. 

Note: Asterisks indicate that the proportion of districts that report having this query capability varies 
significantly by district size (* p < .05 and ** p < .01). 

Source: 2007-08 district survey questions 7 and 8. 



District Supports for Data-driven Decision Making in Schools 

Both the survey and case study data suggest that districts are taking steps to improve the 
capacity of their schools to use data in decision making. 

One of the most commonly reported district policies to encourage schools’ use of data is 
to incorporate this practice into school improvement planning. Sixty-nine percent of districts 
reported requiring all or some of their schools to follow specific data-driven decision making 
practices in formulating their school improvement plans, and 65 percent of districts provide 
teachers with specific processes for how they should use data for instructional purposes. 

The most common district strategies for building school capacity for using data are 
professional development activities, providing support positions for system 
implementation, and the development of tools for generating data and tools for acting on 
data. Over 90 percent of districts responding to the survey reported that they have provided at 
least some school staff with training designed to enhance the school’s capacity to use data in 
ways that improve instruction (e.g., training principals or other school administrators on using 
the data system to analyze student achievement and to provide leadership for data-driven 
decision making in their school). But in many cases training has not been extended to all of the 
district’s schools. Training for teachers on how to use the data system to analyze student 
achievement or on how to use data to change their instructional practices are the two types of 
training least likely to have been given to every school (both provided to all their schools by 53 
percent of districts). 



xii 





Another common district support is providing technical experts in systems, networks or 
databases who can help school staff get access to data from electronic systems. Eighty 
percent of districts say they have provided their schools with this kind of technical expertise, and 
65 percent say they have made such technical expertise available for all of their schools. In 
contrast, making data analysis experts (sometimes called “data coaches”) available to school 
staff is one of the least common supports. Still, 50 percent of districts say they have done this for 
at least some of their schools, and 32 percent say that they have done this for all of their schools. 

Other supports cited by districts in the case study sample are: 

• Providing full- or part-time positions for staff who help teachers work with data, 

• Creating easy-to-read data “dashboards” to help teachers get information from data 
systems that are not user friendly, 

• Developing benchmark and formative assessments to provide teachers with more timely 
data to assess student progress and adjust instruction to meet student needs, and 

• Participating in a range of partnerships to support their efforts to implement data-driven 
decision making in their schools. 

It is important to keep in mind that case study districts were selected for their leadership in 
data use. Survey responses from the national district survey suggest that many districts believe 
that they themselves need examples of good practice in order to guide their schools’ 
implementation of data-driven decision making. The greatest perceived area of need among 
districts is for models of how to connect student data to instructional practice (see Exhibit 
ES-4). Districts want examples of how to identify which practices work best for which students 
and how to adapt instructional strategies to meet the needs of individual students. 
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Exhibit ES-4. District Perceptions of Needed Examples of Good Practice 
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Exhibit reads: In 2007-08, 80 percent of districts reported that they had some or a great need for 
examples of good practice regarding the examination of data to identify which practices work best for 
which students. 



Source: 2007-08 district survey question 24. 



Use of Data Systems and Data-driven Decision Making in Schools 

Site visits suggest that in districts that are leaders in data-driven decision making, the use of 
data in schools is encouraged not through extensive formal professional development but 
rather through ongoing support from colleagues and instructional or data coaches who help 
teachers examine data for their students and develop instructional plans to meet student needs. 
Staff in case study schools described a higher level of support for data-driven decision making 
than did teacher respondents to the national teacher surveys administered in 2005 and 2007. As a 
group, the 36 case study schools appeared to be especially well- supported in the area of funding 
for time for teachers to meet together during the school day to work with data and the provision 
of school-based positions for coaching teachers on how to connect data to instructional 
strategies. 

Even in districts that are actively promoting the use of data, however, school staff 
provided relatively few examples of teachers using data to diagnose areas in which they 
could improve the way they teach. The most common school-level uses of data described by 
teachers and school leaders in the case study sample are school improvement planning, 
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curriculum decisions, and placement or grouping of students for instruction or support services. 
During school site visits, principals, coaches, and teachers were asked to describe specific 
examples of using data to make instructional decisions. From the 36 schools visited in 2007-08, 
188 examples of using data to inform instruction were obtained. Exhibit ES-5 shows the different 
purposes for which school staff described using data and the number of distinct citations of each 
use. The data in Exhibit ES-5 suggest that in districts considered leaders in data-driven decision 
making, most schools are using data to develop goals for school improvement and to do 
curriculum planning. In contrast, school interviewees provided only eight examples of teachers 
using data to determine which aspects of their teaching are working well or poorly. 



Exhibit ES-5. Uses of Data Described by Case Study Schools 



Data Use 


Frequency 


School improvement planning, including setting of quantitative goals 


35 


Curriculum planning based on item or subscale analysis 


25 


Student placement in classes or special services 


22 


Grouping or regrouping of students within a class 


21 


Tailoring instruction to the skill needs of individuals or small groups 


15 


Deciding whether or what to reteach 


13 


Identifying teachers with more successful strategies in order to emulate 
their instructional approach 


11 


Referring students from classroom for supports or services 


9 


Determining what aspects of your teaching are working well/poorly 


8 


Evaluating teacher performance 


7 



Exhibit reads: School improvement planning was the most common data use described by 
staff at the 36 case study schools. 

Source: Case study schools 2007-08. 



In site visit schools, the use of data by teachers to improve their teaching practice 
emerged later than uses such as school improvement planning or student placement. The 

relative frequency of reports of different uses of data within the 36 case study schools suggests 
that different uses emerge over time. Analysts developed three categories of school-level data- 
driven decisions as shown in Exhibit ES-6. The first category covers a range of accountability- 
driven uses of data, the second encompasses matching teaching content to standards or tests and 
giving students adequate time to master the content, and the third category involves using data to 
explore the relative effectiveness of different teaching methods or interventions. 
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Exhibit ES-6. Categories of Data-driven Decision Making 
Decision Type 

Category 1 : Staff examine data for whole grade or school to ascertain areas for school 
improvement; examine data for individual students for purposes of class placement or 
assignment to services, including identifying “bubble kids” whose growth is likely to affect the 
school’s AYP status. 

Category 2: Teachers analyze performance of students in their class on individual items or 
standards for purposes of better aligning their content coverage with the accountability test or 
deciding what to reteach or how to group students within the class. 

Category 3: Staff examine data for different teachers or for different methods dealing with the 
same content to derive insights for improving the way they teach. Staff use comparative data 
to evaluate the effectiveness of specific instructional strategies. 



When the examples of data use described by case study school staff were classified in terms 
of the three data-use categories described in Exhibit ES-6, analysts found that most case study 
schools provided examples of Category 1 and Category 2 uses of data but less than half of all 
schools provided a Category 3 example. Specifically, as shown in Exhibit ES-7, 35 of 36 case 
study schools (97 percent) provided one or more Category 1 examples; 30 of 36 (83 percent) 
provided one or more Category 2 examples; and 17 of 36 (47 percent) provided one or more 
Category 3 examples. 



ES-7. Percent of Case Study Schools Reporting 
Each Decision-making Category 




Data-driven decision making categories 



Exhibit reads: Among case study schools, 97 percent provided examples of 
Category 1 uses of data. 

Source: Case study schools 2007-08. 
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One of the strongest levers that districts can use to increase their schools’ use of data 
systems is to provide timely interim assessment data on those systems. Eighteen of the 36 
schools visited by researchers during school year 2007-08 had been visited a year earlier as part 
of 2006-07 data collection activities. Researchers analyzed site visit reports of examples of 
teacher and school leader data use to ascertain whether or not there were changes in data-use 
practices at those schools between the 2006-07 and 2007-08 school years. Schools in districts 
involved in implementing a system of districtwide interim assessments were more likely than 
other schools to show an increase in data use from year to year, and also provided the most 
striking examples of positive changes in teacher data use practices. In 13 of the 18 schools, there 
was evidence of an increase in data use over this one-year period; in four schools there was no 
indication of a change in the frequency or nature of data use. 2 

Having a set of common assessments that everyone teaching the same content gives to 
their students at about the same time encourages teachers to sit down and share both their 
data and their teaching strategies. When multiple teachers have all given the same recent 
assessment to their students, they can compare their results to identify strengths and weaknesses 
at the class level, something that is not possible if teachers assess different content at different 
times. Researchers found at least some teachers using common assessments and comparing their 
assessment data with each other as a way of comparing and reflecting on their practice in close to 
half of the case study schools. 

Actions that principals can take to encourage teacher use of data include designing and 
implementing regular activities involving the examination of student data and the 
establishment of an organizational climate of trust and mutual respect. Principals encourage 
data use by setting an example through their own activities, designating all or part of teacher 
planning or professional development time as occasions for examining and reflecting on data, 
and communicating expectations around data use. In 25 of the 36 case study schools, researchers 
judged data use to be an important tool in the principal’s assumption of the role of instructional 
leader. Principals at 18 case study schools led whole-school or grade-level meetings dedicated to 
the analysis of data. Principals at several schools maintained schoolwide lists of students who 
were in danger of failing to attain state proficiency standards in order to keep on top of efforts 
being made to support these students during the year. At three of the case study schools 
principals met with teachers individually on a regular basis to discuss their students’ needs and 
the teacher’s plan for the class. 

School staffs’ perceptions of barriers to greater use of data include a sense of lack of 
time , 3 system usability issues, the perception that the data in the system are not useful, and 
district policies around curriculum coverage or pacing that prohibit modifying learning 



2 One school's change in the use of data could not be classified. 

District survey respondents also reported that lack of time for school staff to conduct data-driven decision making 
activities was the greatest barrier to spreading data-driven decision making practices throughout their district (92 
percent of districts cited lack of time as a barrier). 
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time to match student needs. Nationally representative responses on the teacher survey (see 
Exhibit ES-8) suggest that teachers who have access to a data system view their colleagues as a 
resource for data use but have to do much of their work with data on their own time. 



Exhibit ES-8. Teacher Perceptions of the Support 
They Receive for Data Use 
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Exhibit reads: Among teacher survey respondents with access to an electronic data 
system, 71 percent agreed with the statement “I can turn to someone for help.” 
Note: (R) denotes an item that is reverse coded; for these items a disagree 
response indicates more support for data use. 

Source: NETTS teacher survey, 2007. 



At case study schools, staff were much more likely to criticize the quality or timeliness of the 
tests for which data are available in systems than to criticize the usability of the system per se. 

Teacher criticisms of the quality of the data available to them included delay issues, lack of 
alignment with standards, lack of alignment with the school’s instructional approach, and 
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the fact that they received only cross-sectional data rather than longitudinal data for the 
same set of students over time. 

State achievement tests are typically a once-a-year spring event with results unavailable until 
the next fall when students have moved into a new grade level. Staff at many case study schools 
described looking at these data at the time they became available, often using them for school 
improvement planning and for placing students into classes or special services, as noted above. 
Individual teachers look at the prior spring’s scores for the students coming into their class to 
help do planning at the first of the year but soon find more recent and fine-grained information 
from classroom assessments and informal observations guiding their instructional decisions. 

Recommendations 

The surveys and case studies provide a portrait of system and organizational capacity for data 
use in schools and districts. A comparison of this portrait with the ideal of all schools using data 
systems to improve the educational experiences and outcomes for their students suggests a 
number of implications for policies and actions at the school, district, state, and national levels. 
The recommendations below are those of the study team. 

Recommendations for Schools 

• Set clear expectations around the use of student data as the basis for decisions. 

• Integrate collaborative exploration of data into existing structures for joint teacher 
planning and reflection on teaching. 

• Provide a safe environment for teacher examination of their students ’ performance. 

• Support teachers in making the link between data and alternate instructional strategies. 

Recommendations for Districts 

• Think of data-driven decision making as an ongoing systemic process rather than a one- 
time event centered on the acquisition of a data system. 

• Model decision making based on data and present decision-relevant data when 
announcing new policies. 

• Train principals in how to integrate the use of data into school improvement planning 
and promote their teachers ’ use of data for making instructional decisions. 

• Integrate the use of data-driven decision-making practices with district initiatives for 
improving instruction in specific areas. 

• Support time within the work week for teachers to meet with colleagues for planning, 
informed professional development, and data use. 

• Make sure that the district has a data system that gives teachers data that is both timely 
and relevant to their instructioncd decisions. 
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• Provide resources and construct policies so that teachers have access to data relevant to 
the students they are teaching when and where they want it. 

Recommendations for State and National Policy 

• Complement efforts to improve state data systems with investments helping districts 
improve both their data systems and their organizational supports for using data to 
improve instruction. 

• Improve the turnaround time for state assessment data so that schools receive student 
results in time to inform academic-year planning. 

• Promote linkages between local data systems that contain interim assessment data and 
state systems with instructional resources geared to standards. 

• Encourage districts to invest in developing data literacy among district staff in all 
departments. 

• Encourage or require school administrator preparation programs to incorporate 
assessment and data literacy concepts to foster continuous improvement activities 
informed by data as elements of school leadership training. 

• Encourage or require teacher preparation programs to incorporate assessment concepts 
and the use of data for instructional decision making into their teaching methods courses 
(science methods, language arts methods, and so on). 

• Provide districts with good examples of practices that support the development of a data- 
use culture within schools. 
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1. Introduction and Approach 



Proponents of data-driven decision making call on educators to adopt a continuous- 
improvement perspective, with an emphasis on goal setting, measurement, and feedback loops so 
that they can reflect on their programs and processes, relate them to student outcomes, and make 
refinements suggested by the outcome data (CoSN 2004; Dattnow, Park, and Wohlstetter 2007; 
Supovitz and Klein 2003; Wayman, Cho, and Johnston 2007). For example, a 2009 forum of 
state and federal policymakers and education leaders, including U.S. Secretary of Education 
Ame Duncan, emphasized the need to use data systems for continuous improvement (Data 
Quality Campaign 2009b). But until fairly recently, the use of data to make decisions at the 
district, school and classroom levels has been the exception rather than the rule. The lack of 
systems providing user-friendly access to timely, relevant information was one major 
impediment (Wayman 2005). Another was a culture more accustomed to making decisions on 
the basis of educational philosophy or political necessity rather than data (Coburn and Talbert 
2006; Coburn, Toure, and Yamashita 2009). 

In the last decade, however, several forces have converged to make data-driven decision 
making at all levels of the education system a priority. Improved data systems capable of 
tracking individual students’ progress from year to year have become available and have been 
implemented in an increasing number of states and districts (Data Quality Campaign 2008). 
Moreover, some districts are instituting data warehouses that allow them to combine data from 
different systems (for example, student achievement and teacher qualifications) in ways that 
allow investigating education quality issues (Wayman, Stringfield, and Yakimoski 2004). The 
Elementary and Secondary Education Act (ESEA), as reauthorized in 2002, has instituted 
requirements for achievement data reporting by student subgroup, requiring many districts to 
obtain or upgrade their student data systems, and has made schools and districts responsible for 
student achievement. As districts and schools have looked for strategies to help raise 
achievement, the use of data to predict and enhance student performance has emerged as perhaps 
the dominant improvement strategy. Studies of district and school achievement have documented 
a relationship between active use of data and increases in achievement (Datnow, Park, and 
Wohlstetter 2007; Snipes, Doolittle and Herlihy 2002). 

To understand the role of data systems and the supports necessary for teachers to use data 
from any source (electronic and nonelectronic) to inform educational practice, the U.S. 
Department of Education’s Policy and Program Studies Service (PPSS) sponsored a national 
study of Education Data Systems and Decision Making. The study addressed a set of basic 
questions: 

1. What kinds of systems are available to support district and school data-driven decision 
making? Within these systems, how prevalent are tools for generating and acting on data? 

2. How prevalent are organizational supports for school use of data systems to inform 
instruction? 

3. How are school staff using data systems? Do they know how to interpret student data? 
How is school staffs’ use of data systems and of data more broadly influencing 
instruction? 
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This report describes findings from the study’s second round of data collection and analysis, 
conducted during the 2007-08 school year. 



Data Systems: A Prerequisite for Data-driven Decision Making 

For many years, local education agencies (LEAs) have developed and used multiple data 
systems for different purposes (for example, separate systems for finance data, personnel data, 
required accountability information for special education students, school lunch data, enrollment 
and attendance, assessment data). Historically, these data systems were so complex and poorly 
aligned that their use by school staff was not feasible (Wayman and Cho 2009). Moreover, the 
lack of persistent student and teacher identifiers in many of these systems made it impossible to 
obtain a longitudinal view of students’ history within the system or to follow students if they 
transferred schools. Advances in technology and recent policy emphasis on data use have 
resulted in much improved data infrastructures in many districts (Wayman and Cho 2009). 

Beyond Data Systems: Need for a Systemic Approach 

Policymakers envision wide-ranging educational data systems that collect, analyze, and use 
relevant data at every level — from the U.S. Department of Education to the individual teacher 
making instructional decisions for his or her classroom. The mere presence of a student data 
system within a district is a prerequisite but clearly insufficient to bring about change in the way 
that decisions impacting instruction get made. 

Within a school district, many offices (or departments) as well as its schools need to 
participate in order to bring about such a fundamental change in educational decision making 
(Datnow, Park and Wohlstetter 2007; Wayman, Cho, and Johnston 2007). The district needs to 
fund and acquire (or build) a data system capable of storing and manipulating the kinds of 
information needed for educational decisions. Provisions need to be made by the district to 
ensure that intended data system users (district staff, school leaders, instructional coaches, and 
teachers) have ready access to the data in a form they can comprehend and manipulate. 
Assessment and curriculum departments need to make sure that there is up-to-date information 
from assessments linked to the local curriculum available for teacher use. Professional 
development offerings need to include opportunities to learn how to make instructional decisions 
based on data and, if applicable, how to get data out of the system and analyze it to fit one’s 
needs. 

Data-driven Decision-making Framework 

Exhibit 1-1 shows the stages in a data-driven continuous-improvement process: plan, 
implement, assess, analyze data, and reflect (as a precursor to more planning and a refined 
implementation). As the graphic suggests, components of data-driven decision making are part of 
a continuous cycle. The starting point may vary, and there is no fixed end point. 
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Exhibit 1-1. Conceptual Framework for Data-driven Decision Making 




Prerequisites and Supporting Conditions 



1 . Data systems 2. Leadership for 
improvement 
and use of 
data 



3: Tools for 4. Social 

generating structures and 

data time set aside 

for reflection on 
data 



5. Professional 6. Tools for 
development acting on data 
and technical 
support for data 
interpretation 



A major cultural change is required if educators are to make the continuous-improvement 
perspective and the processes of data-driven decision making part of the way in which they 
function. Such a change will not occur without leadership, effort, and well-designed supports. 
The bottom portion of Exhibit 1-1 identifies six major types of prerequisites and supports for 
data-driven decision making that are part of the study’s conceptual framework: 

• State, district, and school data systems 

• Leadership for educational improvement and the use of data 

• Tools for generating actionable data 

• Social structures and supported time for analyzing and interpreting data 

• Professional development and technical support for data interpretation 

• Tools for acting on data 



3 




Data Systems 

ESEA has stimulated an unprecedented level of state activity aimed at improving education 
data systems. Federal requirements for reporting schools’ year-to-year progress in raising the 
percent proficient overall and for specific student categories have led to an examination of 
information system adequacy and the adoption or development of new software systems in many 
states and districts. For example, in 2005, 36 states used unique student identification numbers 
statewide so that students could be followed if they changed districts. In 2008, 48 states did so 
(Data Quality Campaign 2008). As of 2008, state data systems typically include student 
enrollment information, basic demographic data, special program designation (if applicable), and 
scores on state-mandated achievement tests (in most cases, an annual spring testing in language 
arts and mathematics and often a proficiency or “exit” examination required for a high school 
diploma). 

At the local level, it is district systems that are most likely to be used by teachers (U S. 

Department of Education 2009). These systems typically include student scores on state- 
mandated tests, which are obtained from the state or from the state-designated vendor. District 
systems are also likely to include student contact and demographic information, scores on district 
tests, attendance, grades, disciplinary infractions, and course enrollment. 

Leadership for Education Improvement and Use of Data 

Pioneering efforts to promote data-driven decision making within districts and schools have 
found that the active promotion of the effort on the part of the superintendent or principal is vital 
(Marsh, Pane, and Hamilton 2006; Supovitz and Klein 2003; Wayman and Stringfield 2006; 
Young 2006). District and school leaders issue the “call to arms” for improving education and 
using data as a tool to bring about that improvement. Typically, they play a major role in framing 
targets for educational improvement, setting expectations for staff participation in data-driven 
decision making, and making resources, such as supported time, available to support the 
enterprise. 

Tools for Generating Actionable Data 

Increasingly, student achievement data are available at the school level in a form that can be 
disaggregated by student category (ethnicity, free or reduced-price lunch status, special 
education status, etc.). Software systems to support data-driven decision making all generate 
standard student achievement reports, and many also produce custom reports for user-designated 
student groups (an important feature for school staff who want to examine the effects of locally 
developed services for specific student groups). Research indicates, however, that often school 
staff do not find the kinds of data these systems provide particularly useful for guiding 
instruction (Brunner et al. 2005). School staff are frustrated by the fact that the data available to 
them are typically performance on a state achievement test taken six or more months earlier. 
Teachers want up-to-date information on their current group of students, not the students in the 
same grade level the prior year. They also want a greater level of detail concerning individual 
students’ strengths and weaknesses than they can get from standardized test scores (Mandinach 
et al. 2006; Thorn 2002). Although far less common than systems that provide data from prior 
testing, there are examples of systems that produce additional information for decision making 
through tools such as formative assessments that students may take online. In addition, some 
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system designers are working on educational information systems that will integrate data on a 
broad range of transactions, such as daily school attendance, grades, and even library book 
checkouts, with the ultimate goal of automatically recording each interaction a student has with 
the school and student’s assessment and program participation data. 

Social Structures and Supported Time for Analyzing and Interpreting Data 

The most sophisticated data warehouse in the world will have no effect on instruction if no 
one has — or takes — the time to look at the data, reflect on them, and draw inferences for 
instructional planning. Given that time is one of the most basic resources in any organization, 
there need to be strong expectations that administrators will provide time to teachers, and 
educators will take the time to examine data and use it to guide improvements in their programs 
and practices. Such expectations have not been business-as-usual in most schools and districts. 
Case studies of schools active in data-driven decision making suggest that organizational 
structures that include supported time for reviewing and discussing data in small groups greatly 
increase the likelihood that the examination of data will be conducted and will lead to well- 
informed decisions (Copland 2003; Datnow, Park, and Wholstetter 2007; Wayman, Cho, and 
Johnston 2007; Wayman and Stringfield 2006). 

Professional Development and Technical Support for Data Interpretation 

Teacher training generally has not included data analysis skills or data-driven decision- 
making processes in the past (Mandinach et al. 2005; Massell and Goertz 2002). Few 
administrators have this kind of training either. Moreover, the measurement issues affecting the 
interpretation of assessment data — and certainly the comparison of data across years, schools, or 
different student subgroups — are complicated. Data misinterpretation is a real concern (Confrey 
and Makar 2005). For this reason, districts and schools are devoting increasing amounts of 
professional development time to the topic of data-driven decision making. Many argue that the 
practice of bringing teachers together to examine data on their students and relate those data to 
their practices is a valuable form of professional development in its own right (Feldman and 
Tung 2001; Supovitz and Klein 2003; Wayman, Cho, and Johnston 2007). 

Tools for Acting on Data 

The examination of data is not an end in itself but rather a means to improving decisions 
about instructional programs, placements and methods. Once data have been analyzed to reveal 
weaknesses in certain parts of the education program or to identify students who have not 
attained the expected level of proficiency, educators need to reflect on the aspects of their 
processes that may contribute to less-than-desired outcomes and to generate options for 
addressing the identified weaknesses. Some of the data-driven decision-making systems 
incorporate resources that teachers can use in planning what to do differently. These resources 
are typically organized around state content standards and may include lesson plans, instructional 
materials, or descriptions of best practices (Palaich, Good, and van der Ploeg 2004). Resources 
for differentiated instruction can help teachers adapt their instructional approach to students with 
differing strengths and weaknesses. 

Data Sources for the Report 

Findings from this report are drawn from both survey and case study data. The primary data 
set for this report consists of responses of a sample of 529 districts to a survey administered 
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between October 2007 and February 2008. The sampling plan for the district survey was 
developed with the primary goal of providing a nationally representative sample of districts to 
assess the prevalence of district support for data-driven decision making. A secondary goal was 
to provide numbers of districts adequate to support analyses focused on subgroups of districts. 

Of the 529 districts in the final sample of the Study of Education Data Systems and Decision 
Making, 427 responded for a response rate of 81 percent. (Additional information on sampling is 
provided in Appendix A.) 

The case study districts were purposefully selected to include districts that have been active 
in using student data to guide instruction (as a result, these are not typical districts). By focusing 
fieldwork on districts in which many teachers could be expected to be actively looking at student 
data, the study team increased the likelihood of seeing the effects of data use on practice, 
compared with a sample of schools drawn at random. For the 2007-08 case study districts, the 
site selection process was two-staged. The first stage involved the selection of six of the nine 
2006-07 case study districts to be visited for a second round of data collection. The research 
team decided that these districts warranted a second visit because they had been in the process of 
implementing new data systems or activities to support the use of student data at the school level 
during the first round of data collection in 2006-07 and could provide a longitudinal perspective 
on implementation activities. 4 The second stage involved identifying an additional group of six 
districts that have been active in data-driven decision making. These districts were drawn from 
the pool of districts that remained after the initial selection of 10 districts in 2006, supplemented 
by additional districts identified as active data users. For each district visited, respondents 
included key staff involved in the district’s data-informed decision-making activities (e.g., chief 
information officers, directors of curriculum and instruction, directors of research and evaluation, 
directors of accountability, directors of professional development). Within each school, the 
principal, an instructional or data coach (if applicable), and six teachers were interviewed. 

A second survey data set from the U.S. Department of Education’s National Educational 
Technology Trends Study (NETTS) has also been included to provide a national picture of 
teacher perspectives on data use. These data consist of responses of a random sample of K-12 
teachers to a survey administered to 2,509 teachers in spring 2007. The teachers were clustered 
in schools sampled from districts participating in the NETTS study. Teachers were asked to 
report on activities during the 2006-07 school year. 5 

Contents of the Report 

This final report builds on findings from the interim report (U.S. Department of Education 
2009) that described the types of data available to school staff, how school staff use electronic 
data systems, school practices with respect to data-driven decision making, and the supports and 
challenges for school use of student data in planning and implementing instruction. This report 
presents district survey data that provides a national picture of data-driven decision-making 
practices and additional case study data. It also incorporates some of the NETTS teacher survey 



4 For additional information on the district identification process and characteristics of the first set of case study 
districts, see Implementing Data-informed Decision Making in Schools — Teacher Access, Supports and Use (U.S. 
Department of Education 2009). 

5 The response rate for the NETTS teacher survey was 86 percent. 
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data presented previously in the brief entitled Teachers ’ Use of Student Data Systems to Improve 
Instruction-2005 to 2007 (U.S. Department of Education 2007). 

Earlier study reports have documented a dramatic increase in the proportion of teachers with 
access to a student data system between 2005 and 2007. They also described school practices 
with respect to data use and the challenges that are part of student data system implementation. 
In the case study districts visited during the 2006-07 school year, school leaders demonstrated 
their support for school’s use of data by purchasing data systems, modeling data use, and 
providing school-based support positions. Teachers who use data from a student data system do 
so not only on their own but also in collaboration with colleagues. But both teachers and district 
staff members express concerns about teachers’ ability to understand data — many appear to lack 
data literacy skills. 

The data in this final report are primarily descriptive; they do not address the effects of data- 
driven decision making on student outcomes. At the same time, the findings go beyond most 
prior research that has tended to focus on case studies of individual districts and schools to 
provide a broader picture of data-driven decision-making implementation efforts in a dozen 
districts across the country. 6 Study findings lay a foundation on which to build future research 
efforts and will assist policymakers in understanding how data are being used at the local level, 
the conditions affecting use (both positively and negatively), and other issues that arise during 
implementation. 

The remainder of this report is divided into four chapters. Chapter 2 describes the data 
systems that districts are employing to support data collection, storage, retrieval, and analysis; 
the type of data that districts maintain electronically and the ways that districts use data to focus 
on accountability and instructional improvement. Chapter 3 explores district efforts to promote 
and support the use of student data and data systems within their schools, including the 
challenges this effort entails and the strategies that districts are using to overcome obstacles. 
Chapter 4 focuses on school-level implementation of data systems and data-driven decision 
making. Chapter 5 discusses cross-cutting themes and draws implications for policy in this area. 



6 As noted on page 6, the 2006-07 case study sample for the current study included an additional three districts that 
were not revisited in 2007-08, which also helped inform the second year of data collection. 
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2. District Data System Features and District Use of Data 

for Decision Making 



Fundamental to the concept of an information infrastructure to support data-driven decision 
making is the ability to share data across levels of the system (Wayman, Stringfield, and 
Yakimoski 2004). A data system infrastructure is composed primarily of hardware and software 
but also relies on people to gather data, extract the data and use it. Typically, district data 
systems are made up of multiple elements that have a unique role or set of attributes, and the key 
to maximizing the utility of these systems is the ability to integrate and share data files. To get a 
better picture of the information technology available to districts, district survey respondents, 
along with case study sites, were asked by the research team to provide information on the 
electronic student data system or systems that are driving instructional improvement in their 
districts. The study focused on electronic systems that contain data on students but also sought 
information on other electronic systems containing data that might be relevant to instruction (for 
example, data on the professional development received by teachers). The major types of 
electronic student data systems in widespread use are defined in Exhibit 2-1 (Wayman 2005) and 
were used to organize information drawn from the district survey regarding the kinds of data 
systems available to support district and school data-driven decision making (a copy of the 
survey can be found in Appendix B). 



Exhibit 2-1. Types of Electronic Student Data Systems 

1. Student information systems provide real-time access to student data such as attendance, 
demographics, test scores, grades and schedules. 

2. Data warehouses are electronic data collection and storage systems that provide access 
to current and historical data on students, personnel, finance and so on. 

3. Instructional or curriculum management systems provide a unifying framework to support 
access to curriculum and instructional resources such as planning tools, model lesson 
plans, creation of benchmark assessments, linkage to state content or performance 
standards, communication and collaboration tools (e.g., threaded discussion forums). 

4. Assessment systems support rapid organization and analysis of benchmark assessment 
data. 



Two themes emerged from district survey responses and analysis of the case study data: 

• Currently, districts are looking for a way to effectively link their multiple data systems 
since there is no “single solution” data system and there is no simple recipe for effective 
implementation. 

• Districts’ initial acquisition of data systems and use of data has been driven by 
accountability requirements, most recently the ESEA requirements that states and districts 
report on the progress of student subgroups in aggregate toward achieving academic 
proficiency by 2014 and to close the proficiency gap among student subgroups. 
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A Profile of District Data Systems 

To provide an overview or snapshot of the electronic data systems that U.S. school districts 
employ and those systems’ capabilities, responses from multiple survey items were combined 
into a handful of key data system elements to assess the extent to which district data systems are 
providing information to support better decisions about instruction (see Data Quality Campaign 
2008 for a similar approach to depicting state data systems). Data systems with a greater variety 
of data and tools that support looking at the performance of students with different educational 
experiences are much better equipped to support the kind of inquiry necessary for continuous 
improvement activities — the ongoing examination of student data to assess the effectiveness of 
education activities and to refine programs and practices to improve student outcomes. 

Analysts organized survey responses to examine ten elements of quality for district data 
systems (Exhibit 2-2): 

• Student information system 

• Data linkages 

• Instructional or curriculum management system 

• Assessment system 

• Student performance data to measure academic growth 

• Student-level enrollment, demographic and program participation information 

• Teacher-level data 

• Student-level graduation, post-graduation and dropout data 

• College readiness data 

• Data quality assurance 

The profile provides information on the percentage of districts with all of the subelements 
present for a particular element (i.e., districts responded positively to all of the survey items 
comprising that element). It was fairly unusual for districts to have all of the subelements. While 
65 percent of districts have all the subelements for a student information system only 44 percent 
have all the subelements for student performance data to measure academic growth. 

Each element score in Exhibit 2-2 (right-hand column) represents the median number of 
subelements present across districts. The number in parentheses represents the percentage of 
districts having at least the median number of subelements in place. So, for example, when it 
comes to linking data, 64 percent of districts have the capacity to conduct two out of the five 
subelement linkages that make up this element (primarily student performance linked to AYP 
subgroups and linked to specific teachers). Seventy-four percent of districts report having one 
out of the two subelements under teacher-level data (primarily teacher qualifications). 
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Exhibit 2-2. Profile of District Data System Elements in 2007-08 











Median No. of 
Subelements 






Percent of 


Percent of 


Across All 






Districts 


Districts 


Districts (Percent 






With 


With All 


of Districts With at 




Data System Element 


Subelement 


Subelements 


Least Median No.) 


1 . 


Student information system (3 subelements) 




65 


3 

( 65 ) 




Have system (Q4a) 


100 








Ability to generate standard accountability reports or 
district report card and school report cards (Q8a) 


66 








Transaction capture (Q8b) 


92 






2. 


Data linkages (5 subelements) 




15 


2 

( 64 ) 




Linking school performance and finance data (Q7g) 


24 








Student performance linked to teacher information or 
characteristics (Q7c) 


36 








Student performance linked to AYP subgroups (Q7a) 


65 








Student performance linked to specific teachers (Q7b) 


64 








Student performance linked to specific instructional 
programs (Q7d) 


40 






3. 


Instructional/curriculum management system (2 




47 


1 




subelements) 






( 79 ) 




Have system (Q4c) 


64 








Links to curricular resources (Q8f) 


62 






4. 


Assessment system (2 subelements) 




43 


1 

( 81 ) 




Have system (Q4d) 


79 








Assessments available in reading, mathematics, or 
other core subject areas that students take online 
(Q8d) 


46 






5. 


Student performance data to measure academic 




44 


3 




growth (4 subelements) 






( 71 ) 




Student test scores on statewide assessments (Q5a) 


93 








Student test scores on district-administered 
assessments (Q5b) 


72 








Drill-down capability (Q8c) 


72 








Individual student assessment performance over time 
(Q7e) 


70 






6. 


Student-level enrollment, demographic and 




59 


8 




program participation information (8 subelements) 






( 59 ) 




Student grades (Q5e) 


95 








Student course enrollment histories (Q5f) 


92 








Prior school(s) attended within the district (Q5h) 


86 








Student demographics (Q5g) 


98 








Student attendance (Q5k) 


98 








Student behavior (Q5I) 


87 








Student special education information (Q5i) 


84 








Individual student history over time (Q7f) 


81 







Exhibit 2-2 continues on next page 
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Exhibit 2-2. Profile of District Data System Elements in 2007-08 (continued) 





Data System Element 


Percent of 
Districts By 
Subelement 


Percent of 
Districts 
With All 
Sublements 


Median No. of 
Subelements 
Across All Districts 
(Percent of 
Districts With at 
Least Median No.) 


7. 


Teacher-level data (2 subelements) 




47 


1 

( 74 ) 




Teacher qualifications (Q5p) 


73 








Teacher professional development (Q5q) 


47 






8. 


Student-level graduation, post-graduation and 
dropout data (3 subelements) 




38 


2 

( 86 ) 




Differential codes for students no longer enrolled 
(Q5m) 


93 








Student graduation status (Q5n) 


90 








Student status after graduation (Q5o) 


34 






9. 


College readiness (1 subelement) 




57 


1 

( 57 ) 




Student test scores on SAT, ACT, and Advanced 
Placement tests (Q5d) 


57 






10. 


Assessment of data quality (3 subelements) 




46 


2 

( 83 ) 




District/state has disseminated data collection 
guidelines & recommended data information 
management and security practices to schools (Q12) 


71 








District has staff or outside source responsible for 
receiving & preparing files from outside sources to 
load into the student data system (Q13) 


89 








Greater than 90% of data captured by the district’s 
student data system(s) that drive instructional 
improvement are accurate (Q14) 


65 







Exhibit reads: In 2007-08, 65 percent of districts with electronic data systems reported having all three of 
the subelements for student information systems (whereas 100 percent reported having the first 
subelement). The median number of subelements present is three. 

Source: 2007-08 district survey questions 4, 5, 7, 8, 12-14. 



The profile suggests that district access to robust data systems is still limited. Such 
limitations may impede district achievement of three of the education goals outlined in the 
American Recovery and Reinvestment Act (ARRA) of 2009: (a) establishing pre-K to college and 
career data systems that track student progress, (b) providing and assessing effective 
interventions for the lowest-performing schools, and (c) assessing teacher effectiveness and the 
equitable distribution of qualified teachers for all students (particularly students who are most in 
need). 

As was true in prior years, districts are maintaining more administrative data than data 
that are targeted to individual student performance (U.S. Department of Education 2009). 
Fifty-nine percent of districts have all eight of the subelements under student-level enrollment, 
demographic and program participation information (element 6), and 65 percent have all three 
subelements associated with a student information system (element 1), whereas only 44 percent 
have all four subelements for student performance data to measure academic growth (element 5). 



12 





An even larger impediment appears to be districts’ inability to link data across systems 
(element 2), with just 15 percent of districts capable of all of the linkages described by the 
subelements. The fact that only 15 percent of districts report having all five subelements 
related to data linkages points to the challenges in achieving the efficiencies that technology 
was envisioned to offer. Lack of interoperability between systems complicates data analysis, 
and differences in system interfaces increase training requirements for district and school staff. 

As discussed later in this chapter, districts are using multiple data systems and over 60 percent 
reported that lack of interoperability across their data systems was a current barrier to expanded 
use of data driven decision making. 

Data quality is of particular concern to policymakers given their focus on accountability, but 
the issue of data quality is part of a broader context. The district data system profile indicates that 
the average district carries out two of the three activities associated with assessing data quality 
(element 10). About two-thirds of districts (65 percent) reported that 90 percent or greater of the 
data captured by their student data system or other system that drives instructional improvement 
is accurate. Case study and survey respondents indicated that data with the biggest accuracy 
problems tend to be items such as student demographics that rely on self-report or are provided 
by parents or else data that often changes frequently (e.g., student schedules, students receiving 
tutoring outside the school day). The case studies suggested that the concept of data accuracy is 
subject to interpretation and is more nuanced than the statistic based on survey responses may 
suggest. Data for accountability purposes (e.g., state test scores) have consequences attached, 
and therefore accuracy is critical. Data used for ongoing adjustments to instructional practice 
tend to be generated more frequently, with any one data entry having less serious consequences, 
justifying a lower level of effort for quality control. 

The remainder of this chapter provides a more indepth look at the components that make up 
district data systems, where these components come from, the types of data maintained in district 
data systems, the tools available to carry out more sophisticated analyses, and how districts use 
their data systems. 

District Data System Features 

As discussed in the introduction, the standards movement and state and federal accountability 
requirements have placed a greater emphasis on using data to monitor progress. Advances in 
technology have made it possible to link multiple datasets, to track change over time, and engage 
in much more sophisticated analysis activities. But there is often a gap between what is 
technically possible and what districts are equipped to do. One of the study’s goals was to 
ascertain the extent to which districts have access to electronic data systems with the capacity to 
realize these benefits. To this end, district survey respondents were asked to report on the 
elements of their current student data system. The application of sampling weights to the district 
survey data produced a nationally representative portrait of the educational data systems used by 
districts in school year 2007-08. 

Very few districts responding to the survey (10 out of 427) indicated that they did not 
have any kind of electronic student data system or tools to enhance educational decision 
making. The ten districts without electronic data systems tend to be smaller than other districts 
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